Also sync content into Azure File Share and R2 buckets #745

lemeurherve · 2023-10-09T14:08:01Z

This PR allows synchronizing the content of updates.jenkins.io to the new Azure File Share and CloudFlare R2 buckets I've put in place for the migration of this service from the current VM to an high availability service on publick8s.

This migration will allow us to reduce our AWS cost by a third, thanks to the free outbound bandwidth of CloudFlare R2.

It will also allow us to benefit from a distributed service thanks to mirrorbits, and (later) the use of a bucket in each of the 6 regions offered by CloudFlare for its buckets: https://developers.cloudflare.com/r2/reference/data-location/#available-hints.

For this sync I'm using the aws-cli and azcopy tools now installed on trusted agent since jenkins-infra/jenkins-infra#3099

For aws-cli, it's using a non default updates-jenkins-io profile, stored in /home/jenkins/.aws on the agent.

For azcopy, it's using an account SAS token, stored manually as global credentials on trusted.ci.jenkins.io.
We're not reusing blobxfer, which authenticates with a global storage account access key, less secure than the SAS token which is restricted to a specific file share, and later only on the agent IP address. (IP restriction disabled for now, will be restored later with the workaround mentioned in jenkins-infra/azure#496)

Test job, using a copy of the content of updates.jenkins.io instead of running generate.sh script as we don't have access in this test to the 400Go cache folder:
https://trusted.ci.jenkins.io:1443/job/update_center_test_lemeurherve_helpdesk2649/job/test-update-center-pr-745/

If this sync works as expected, the previous one in the script targeting the current VM will be removed, as the flag mechanism used to optionally sync azure.updates.jenkins.io.

Ref: jenkins-infra/helpdesk#2649

dduportal

LGTM!

I've added a few codestyle proposal (2 are shellcheck not passing) but it's is ready to roll for me.

We need 2 other distincts reviews than mine to be sure we do not miss anything

site/publish.sh

lemeurherve · 2023-10-09T15:44:01Z

Putting the PR in draft while working on the following error:

Failed with error service level URLs (https://updatesjenkinsio.file.core.windows.net) are not supported in sync:

lemeurherve · 2023-10-09T16:08:53Z

Found the culprit!

https://updatesjenkinsio.file.core.windows.net/: service level URL, not working.

https://updatesjenkinsio.file.core.windows.net/updates-jenkins-io: file share URL, working as intended.

I'm updating the corresponding credentials on trusted.ci.jenkins.io, and will put the PR in ready for review afterward.

site/publish.sh

lemeurherve · 2023-10-18T09:08:20Z

Work in progress on this branch for now: https://github.com/jenkins-infra/update-center2/blob/pr-745/site/publish.sh

I'll report the changes here when the tests will be conclusive.

smerle33 · 2023-10-25T07:28:12Z

we now need to launch a mirrorbits scan ID or mirrorbits scan -all (Scan all mirrors at once) through kubectl to trigger a scan after rsync/azsync/awssync

dduportal · 2023-10-26T17:29:06Z

we now need to launch a mirrorbits scan ID or mirrorbits scan -all (Scan all mirrors at once) through kubectl to trigger a scan after rsync/azsync/awssync

Update: kubectl is now installed

site/publish.sh

… R2 buckets

Co-authored-by: Damien Duportal <damien.duportal@gmail.com>

lemeurherve · 2023-11-20T15:08:25Z

@daniel-beck @Wadeck @MarkEWaite @timja this PR is ready for review.

I've put every new operations behind an opt-in flag, to be set to optin as env var on the job to sync both updates.jenkins.io and azure.updates.jenkins.io.

After approval and before merging, we intend to make a backup of updates.jenkins.io JSONs from pkg.origin.jenkins.io VM (maybe a daily one), and a snapshot of the disk where the update-center2 job cache is stored.

lemeurherve · 2024-01-25T11:26:55Z

The only change expected to this PR is the addition of an az login with a service principal followed by an az storage share generate-sas so the SAS token used to manipulate the file share will be short lived and revocable.

Ref:

[INFRA-3100] Migrate updates.jenkins.io to another Cloud helpdesk#2649 (comment)

Check if we could replace blobxfer by azcopy helpdesk#3414 (comment)

lemeurherve · 2024-03-07T17:39:13Z

The only change expected to this PR is the addition of an az login with a service principal followed by an az storage share generate-sas so the SAS token used to manipulate the file share will be short lived and revocable.

The change will be similar to jenkins-infra/crawler#144, tested on trusted.ci.jenkins.io with our PoC job in the pr-745 branch:

update-center2/site/publish.sh

Lines 56 to 59 in d799620

    
           # shellcheck source=/dev/null 
        
           fileShareSignedUrl=$(source ./site/get-fileshare-signed-url.sh) 
        
           # Sync Azure File Share content using www3 to avoid symlinks 
        
           time azcopy sync "${ROOT_FOLDER}/www3/" "${fileShareSignedUrl}" \

https://trusted.ci.jenkins.io:1443/job/update_center_test_lemeurherve_helpdesk2649/job/test-update-center-pr-745/153/console

Sourcing the script won't be needed with jenkins-infra/jenkins-infra#3323 as it will be directly executable from /usr/local/bin, in the $PATH.

I'm waiting for jenkins-infra/jenkins-infra#3323 to be approved and merged before pushing it here so this PR remains valid in the mean time.

lemeurherve · 2024-03-08T11:18:44Z

I'm waiting for jenkins-infra/jenkins-infra#3323 to be approved and merged before pushing it here so this PR remains valid in the mean time.

Corresponding change pushed: ebfac65

I need to add an AzureServicePrincipal credentials to the update-center2 freestyle job, with the expected environment variable names like https://github.com/jenkins-infra/crawler/blob/a846b4e4ef7088f732c25e369ef56ab05bf1e77e/Jenkinsfile#L74-L77

I've also fixed the kubectl call by updating mirrorbits pod name pattern in 6261823

lemeurherve · 2024-03-08T12:13:15Z

I need to add an AzureServicePrincipal credentials to the update-center2 freestyle job, with the expected environment variable names like https://github.com/jenkins-infra/crawler/blob/a846b4e4ef7088f732c25e369ef56ab05bf1e77e/Jenkinsfile#L74-L77

Credentials configured in update_center job on trusted.ci.jenkins.io.

This pull request is complete and mergeable, ready for review.

lemeurherve requested review from MarkEWaite, dduportal, daniel-beck, Wadeck, smerle33 and a team October 9, 2023 14:08

dduportal approved these changes Oct 9, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

site/publish.sh Outdated Show resolved Hide resolved

site/publish.sh Outdated Show resolved Hide resolved

lemeurherve commented Oct 9, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

daniel-beck changed the title ~~feat(udpates.jenkins.io): also sync content into Azure File Share and R2 buckets~~ Also sync content into Azure File Share and R2 buckets Oct 9, 2023

lemeurherve requested a review from timja October 9, 2023 14:41

lemeurherve commented Oct 9, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

lemeurherve marked this pull request as draft October 9, 2023 15:43

dduportal approved these changes Oct 10, 2023

View reviewed changes

dduportal reviewed Oct 10, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

smerle33 mentioned this pull request Oct 25, 2023

feat(trusted/agent): add kubectl cli command jenkins-infra/jenkins-infra#3137

Merged

smerle33 mentioned this pull request Oct 27, 2023

feat (mirrorbits-parents): add service account jenkins-infra/helm-charts#897

Merged

lemeurherve force-pushed the helpdesk2649-upload-to-azure-updates-jenkins-io branch 2 times, most recently from ae75b1a to 11b661c Compare November 17, 2023 13:13

dduportal reviewed Nov 17, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

site/publish.sh Outdated Show resolved Hide resolved

site/publish.sh Outdated Show resolved Hide resolved

site/publish.sh Outdated Show resolved Hide resolved

lemeurherve force-pushed the helpdesk2649-upload-to-azure-updates-jenkins-io branch 2 times, most recently from a6d9a3f to c9f636a Compare November 17, 2023 15:01

dduportal reviewed Nov 17, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

dduportal reviewed Nov 17, 2023

View reviewed changes

site/publish.sh Outdated Show resolved Hide resolved

dduportal approved these changes Nov 20, 2023

View reviewed changes

lemeurherve force-pushed the helpdesk2649-upload-to-azure-updates-jenkins-io branch from 7a7348c to ec5cc0c Compare November 20, 2023 13:58

lemeurherve and others added 8 commits November 20, 2023 15:00

feat(udpates.jenkins.io): also sync content into Azure File Share and…

9fce278

… R2 buckets

parrallilization

3aca271

add a mirrorscan step at the end of the script

df5d525

cleanup

7a8a255

fix typo

87b9814

array of R2 buckets

70c1d91

move new operations to the opt-in block

0f75f15

put mirrorbits triggering behind the opt-in flag

c15b967

lemeurherve force-pushed the helpdesk2649-upload-to-azure-updates-jenkins-io branch from ec5cc0c to 2822a48 Compare November 20, 2023 14:02

Apply suggestions from code review

f3c6819

Co-authored-by: Damien Duportal <damien.duportal@gmail.com>

lemeurherve force-pushed the helpdesk2649-upload-to-azure-updates-jenkins-io branch from 2822a48 to f3c6819 Compare November 20, 2023 14:40

lemeurherve marked this pull request as ready for review November 20, 2023 15:03

timja approved these changes Nov 20, 2023

View reviewed changes

fixup add missing azcopy exclude-path arg

189e6d0

daniel-beck added the wrapper PR affects wrapper behavior and will be effective immediately when merged. label Mar 7, 2024

lemeurherve added 2 commits March 7, 2024 20:44

mirrorbits pod name doesn't contain "-lite" anymore

6261823

short-lived SAS token by using get-fileshare-signed-url.sh

ebfac65

daniel-beck merged commit 82c4045 into jenkins-infra:master Mar 12, 2024
2 checks passed

lemeurherve deleted the helpdesk2649-upload-to-azure-updates-jenkins-io branch March 12, 2024 10:21

lemeurherve mentioned this pull request Mar 19, 2024

[INFRA-3100] Migrate updates.jenkins.io to another Cloud jenkins-infra/helpdesk#2649

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Also sync content into Azure File Share and R2 buckets #745

Also sync content into Azure File Share and R2 buckets #745

lemeurherve commented Oct 9, 2023 •

edited

Loading

dduportal left a comment

lemeurherve commented Oct 9, 2023

lemeurherve commented Oct 9, 2023 •

edited

Loading

lemeurherve commented Oct 18, 2023

smerle33 commented Oct 25, 2023 •

edited

Loading

dduportal commented Oct 26, 2023

lemeurherve commented Nov 20, 2023

lemeurherve commented Jan 25, 2024 •

edited

Loading

lemeurherve commented Mar 7, 2024 •

edited

Loading

lemeurherve commented Mar 8, 2024

lemeurherve commented Mar 8, 2024

Also sync content into Azure File Share and R2 buckets #745

Also sync content into Azure File Share and R2 buckets #745

Conversation

lemeurherve commented Oct 9, 2023 • edited Loading

dduportal left a comment

Choose a reason for hiding this comment

lemeurherve commented Oct 9, 2023

lemeurherve commented Oct 9, 2023 • edited Loading

lemeurherve commented Oct 18, 2023

smerle33 commented Oct 25, 2023 • edited Loading

dduportal commented Oct 26, 2023

lemeurherve commented Nov 20, 2023

lemeurherve commented Jan 25, 2024 • edited Loading

lemeurherve commented Mar 7, 2024 • edited Loading

lemeurherve commented Mar 8, 2024

lemeurherve commented Mar 8, 2024

lemeurherve commented Oct 9, 2023 •

edited

Loading

lemeurherve commented Oct 9, 2023 •

edited

Loading

smerle33 commented Oct 25, 2023 •

edited

Loading

lemeurherve commented Jan 25, 2024 •

edited

Loading

lemeurherve commented Mar 7, 2024 •

edited

Loading